Input Prioritization for Testing Neural Networks

Deep neural networks (DNNs) are increasingly being adopted for sensing and control functions in a variety of safety and mission-critical systems such as self-driving cars, autonomous air vehicles, medical diagnostics, and industrial robotics. Failures of such systems can lead to loss of life or property, which necessitates stringent verification and validation for providing high assurance. Though formal verification approaches are being investigated, testing remains the primary technique for assessing the dependability of such systems. Due to the nature of the tasks handled by DNNs, the cost of obtaining test oracle data---the expected output, a.k.a. label, for a given input---is high, which significantly impacts the amount and quality of testing that can be performed. Thus, prioritizing input data for testing DNNs in meaningful ways to reduce the cost of labeling can go a long way in increasing testing efficacy. This paper proposes using gauges of the DNN's sentiment derived from the computation performed by the model, as a means to identify inputs that are likely to reveal weaknesses. We empirically assessed the efficacy of three such sentiment measures for prioritization---confidence, uncertainty, and surprise---and compare their effectiveness in terms of their fault-revealing capability and retraining effectiveness. The results indicate that sentiment measures can effectively flag inputs that expose unacceptable DNN behavior. For MNIST models, the average percentage of inputs correctly flagged ranged from 88% to 94.8%

Manifold-based Test Generation for Image Classifiers

Neural networks used for image classification tasks in critical applications must be tested with sufficient realistic data to assure their correctness. To effectively test an image classification neural network, one must obtain realistic test data adequate enough to inspire confidence that differences between the implicit requirements and the learned model would be exposed. This raises two challenges: first, an adequate subset of the data points must be carefully chosen to inspire confidence, and second, the implicit requirements must be meaningfully extrapolated to data points beyond those in the explicit training set. This paper proposes a novel framework to address these challenges. Our approach is based on the premise that patterns in a large input data space can be effectively captured in a smaller manifold space, from which similar yet novel test cases---both the input and the label---can be sampled and generated. A variant of Conditional Variational Autoencoder (CVAE) is used for capturing this manifold with a generative function, and a search technique is applied on this manifold space to efficiently find fault-revealing inputs. Experiments show that this approach enables generation of thousands of realistic yet fault-revealing test cases efficiently even for well-trained models

Synthesis and Properties of Fluorinated Polyimides from Rigid and Twisted Bis(Trifluoromethyl)Benzidine for Flexible Electronics

Fluorinated polyimides were prepared from the twisted benzidine monomer containing two trifluoromethyl (CF3) groups on one aromatic ring. The diamine monomer having a rigid and nonplanar structure was polymerized with typical dianhydride monomers including BPDA, BTDA, ODPA, 6-FDA, and PMDA, to obtain the corresponding polyimides. Most polyimides are soluble in organic solvents due to their twisted chain structure and can be solution cast into flexible and tough films. These films have a UV-vis absorption cut-off wavelength at 354–398 nm and a light transparency of 34–90% at a wavelength of 550 nm. They also have tensile strengths of 92–145 MPa and coefficients of thermal expansion (CTE) of 6.8–63.1 ppm/°C. The polymers exhibited high thermal stability with 5% weight loss at temperatures ranging from 535 to 605°C in nitrogen and from 523 to 594°C in air, and high glass temperature (Tg) values in the range of 345–366°C. Interestingly, some of the soluble polyimides showed thermo-responsive behaviors in organic solvents presumably due to the multiple hydrogen bondings with unsymmetrically positioned two CF3 groups along the polymer chains

Soluble Poly(amide-imide)s from Diamide–Diamine Monomer with Trifluoromethyl Groups

A series of soluble aromatic poly(amide-imide)s (PAIs) was prepared from a new diamide–diamine monomer having biphenyl units with two CF3 groups. The diamide–diamine monomer was polymerized with 2,2′-bis(trifluoromethyl)benzidine and pyromelltic dianhydride through an imidization reaction to prepare PAIs with a controlled imide/amide bond ratio in the main chains. While the PAIs with the highest imide bond content showed a limited solubility, other PAIs were soluble in polar organic solvents and can be solution-cast into flexible freestanding films. All PAIs exhibited high thermal stability with 5% weight loss temperature (Td5) from 464 to 497 °C in air, and no appearance of glass transition up to 400 °C. Notably, the linear coefficient of thermal expansion (CTE) value of the PAI films was linearly decreased with the imide bond content and varied from 44.8 to 7.8 ppm/°C

Toward Rigorous Object-Code Coverage Criteria

Object-branch coverage (OBC) is often used as a measure of the thoroughness of tests suites, augmenting or substituting source-code based structural criteria such as branch coverage and modified condition/decision coverage (MC/DC). In addition, with the increasing use of third-party components for which source-code access may be unavailable, robust object-code coverage criteria are essential to assess how well the components are exercised during testing. While OBC has the advantage of being programming language independent and is amenable to non-intrusive coverage measurement techniques, variations in compilers, and the optimizations they perform can substantially change what is seen as an object branch, which itself appears to be an informally understood concept. To address the need for a robust object coverage criterion, this paper proposes a rigorous definition of OBC such that it captures well the semantic of source code branches for a given instruction set architecture. We report an empirical assessment of these criteria for the Intel x86 instruction set on several examples from embedded control systems software. Preliminary results indicate that object-code coverage can be made robust to compilation variations and is comparable in its bug-finding efficacy to source level MC/DC

Contract discovery from black-box components

Associated research group: Critical Systems Research GroupComplex computer-controlled systems are commonly constructed in a middle-out fashion where existing subsystems and available components have a significant influence on system architecture and drive design decisions. During system design, the architect must verify that the components, put together as specified in the architecture, will achieve the desired system behavior. This typically leads to further design modifications or adjustments to requirements triggering another iteration of the design-verify cycle. For software components that are acquired from third-parties, often the only definitive source of information about the component's system-relevant behavior -- its contract -- is its object code. We posit that existing static and dynamic analysis techniques can be used to discover contracts that can help the system designer and specifically discuss how symbolic execution of object code may be particularly well-suited for this purpose

Highly Transparent Aromatic Polyamides from Unsymmetrical Diamine with Trifluoromethyl Groups

Soluble and transparent wholly aromatic polyamides (PAs) were synthesized from an unsymmetrical diamine monomer having trifluoromethyl (CF3) groups, 4-(4′-aminophenoxy)-3,5-bis(trifluoromethyl)aniline. The monomer was polymerized with several dicarboxylic acid monomers via the Yamazaki–Higashi polycondensation method. All of the synthesized polyamides have an amorphous morphology, and they are soluble in many polar organic solvents at room temperature. Flexible and transparent films of the polyamides were prepared by solution casting and these polymer films show good optical transparencies with cut-off wavelengths of 337–367 nm and transparencies of 88%–90% at 550 nm. In addition, all the polymers were thermally stable over 400 °C and exhibited glass transition temperatures (Tg) higher than 300 °C. Unsymmetrically inserted trifluoromethyl groups on polyamides improves the solubility as well as the transparency of the polymers while maintaining good thermal properties. They also showed low refractive indices around 1.5333~1.5833 at 633 nm owing to the existence of low polarizable trifluoromethyl groups